Running Local LLMs on Linux: From 8B to 120B Parameters, with OpenCode and Claude

01Introduction: Why Local LLMs Matter Now

By the end of this guide you will have a language model running on your own Linux machine and connected to two coding agents, OpenCode and Claude Code. You will also know which model fits the hardware you already own, what the next hardware step costs in October 2026, and when paying for Claude's API is still the cheaper option.

Running a model locally means the weights sit on your disk and your GPU or CPU computes every token. A prompt goes nowhere unless you send it somewhere. Two changes this year made the setup far more practical. In January, Ollama 0.14 added Anthropic Messages API compatibility, and LM Studio, vLLM and llama.cpp now ship their own /v1/messages endpoints, so Claude Code can talk to a local server after you set three environment variables. Small models also improved. Alibaba reports that Qwen3.8-27B scores 61.7 on SWE-bench Pro when run inside the Claude Code harness, and Ollama packages it as an 18 GB download.

Hardware is the constraint. A memory shortage has pushed GPU and unified-memory prices far above their launch levels. The RTX 5090 launched at $1,999, yet its lowest new asking price in the US was $4,920.99 on 11 October, and the cheapest card in stock in India costs about ₹7.49 lakh. Every tier in this guide carries a current price so you can decide with real numbers rather than launch-day ones.

The sections below cover six model families from 8B to 120B parameters, five runtimes, the OpenCode and Claude Code setups, and six tutorials split into basic, intermediate and advanced levels. Five small tools run in the page itself: a memory calculator, a break-even calculator, a model picker, a config generator and a tutorial filter. None of them sends data anywhere. Commands target Ubuntu and Debian, with Fedora and Arch notes only where a command changes.

02Advantages, Trade-Offs And Cost Savings

These are the advantages that hold up in practice:

• Privacy and data residency: source code, contracts and customer records stay on hardware you own, and no third-party processor sits in the request path.
• Offline use: a laptop on a train or a machine in an air-gapped lab behaves exactly as it does on the office network.
• No rate limits: a batch job runs as fast as your GPU allows, with no HTTP 429 responses and no quota resets to wait for.
• Predictable cost: the hardware is a one-off purchase, and electricity is the only running expense.
• Latency: there is no network hop, so time to first token depends only on your machine and the prompt length.
• Customisation: you can fine-tune, change chat templates, pick a quantization and keep one model version for years.
• Vendor independence: a downloaded Apache 2.0 model such as Gemma 4, gpt-oss or Qwen3.8-27B cannot be withdrawn or repriced.
• Compliance evidence: an auditor can inspect the exact model file and server version that produced an output.

Open models still trail the frontier. In Pinggy's October 2026 roundup of Artificial Analysis scores, Claude Opus 5.5 leads at 58, the best open-weight model scores 46, and Qwen3.8-27B, the strongest model that fits one 24 GB card, scores 34. Long agentic sessions expose that gap first, because small errors compound over dozens of tool calls. Maintenance also becomes your job: drivers, CUDA versions, context settings and model updates all need attention.

A Worked Break-Even Example

Take a team of three developers running agentic coding sessions on Claude Sonnet 5.5, which Anthropic's pricing page lists at $2 input and $10 output per million tokens. Agent loops resend a lot of context, so assume 300 million input tokens and 30 million output tokens a month. That is 300 × $2 + 30 × $10 = $900 before prompt caching; cache reads cost $0.10 per million on Sonnet 5.5, so a well-cached workload pays less.

Now buy a 128 GB Framework Desktop for $3,449 and spread it over 36 months, which is $95.81 a month. At an average 200 W for eight hours a day, it draws 48 kWh a month. At the EIA's 2026 US residential average of 18.2 cents per kWh, that is $8.74. In India, using an assumed ₹8 per kWh for a household in a higher consumption slab, it is ₹384. Check your own DISCOM schedule, because slabs vary by state and usage.

A local model will not replace Sonnet on every task, so assume it absorbs 60 percent of the tokens. The cloud bill falls to $360 and the team saves $540 a month. The break-even formula is:

months to break even = hardware cost ÷ (monthly cloud savings − monthly electricity)

In the US that gives $3,449 ÷ ($540 − $8.74) = 6.5 months. In India the electricity term is smaller, so the payback is about the same, although import duty raises the hardware price. Run the same arithmetic on Claude Haiku 5.5, priced at $0.10 input and $0.50 output for prompts up to 100,000 tokens, and the picture flips. The whole 330-million-token workload costs $45 a month, a 60 percent shift saves $27, and the box would need more than 15 years to pay for itself. For cheap, short prompts, buy local hardware for privacy or offline needs, not to save money.

The hybrid split is the pattern this guide recommends: local models handle private, bulk and routine work, and Claude handles the hardest reasoning and the longest agentic runs. Tutorial 6 builds that split with an MCP server and a router. Try your own numbers below.

03Six Model Families From 8B To 120B

Two terms come up in every row of the table below. A dense model runs every parameter for every token. A mixture-of-experts (MoE) model sends each token through a few expert blocks, so its active parameter count sets the compute per token while its total count still sets the memory it needs. Quantization stores weights in fewer bits: Q4_K_M is a popular 4-bit GGUF format for llama.cpp-based tools, and MXFP4 is the 4-bit format that gpt-oss was trained in.

Qwen

Alibaba released the open weights of Qwen3.8-27B on Hugging Face in August 2026 under Apache 2.0. It is a 27B dense vision-language model with a native context of 262,144 tokens that extends to about one million with YaRN scaling. It thinks before answering by default, and a reasoning_effort setting trades depth for speed. The qwen3.8 entry in Ollama's library lists tools, thinking and vision support at 18 GB. For agentic coding on one 24 GB card it is the default pick in this guide.

Gemma

Google launched Gemma 4 under an Apache 2.0 licence on 2 April 2026. The family includes E2B and E4B edge models, a 26B MoE that activates 3.8B parameters per token and a 31B dense model, and Ollama also carries a 12B tag. The workstation sizes take 256K tokens of context, support native function calling and accept images. The 26B MoE generates faster because so few parameters run per token, while the 31B dense model keeps more quality.

Mistral

Mistral covers both ends of the ladder under Apache 2.0. The Ministral 3 8B model card lists an 8.4B language model plus a 0.4B vision encoder, native function calling and a 256K context window, and says the model fits in 12 GB of VRAM in FP8. A 14B sibling follows the same design. At the top, Mistral Small 4 on Hugging Face is a 119B MoE with 6.5B active parameters, 128 experts and a 256K context, and it merges chat, reasoning and agentic coding into one model.

Llama

Meta's Llama 3.3 70B Instruct is the dense 70B option. It has a 128K context, eight officially supported languages including Hindi, and tool calling. Its licence is the Llama 3.3 Community License, which requires "Built with Llama" attribution and sets a 700-million-monthly-user threshold, so read it before shipping a product. At Q4_K_M it needs about 42 GB for weights alone, which puts it on a dual-GPU build or a 128 GB unified-memory machine.

gpt-oss

OpenAI's two open models ship under Apache 2.0. The gpt-oss-120b model card gives 117B total and 5.1B active parameters for the large model and 21B total and 3.6B active for gpt-oss-20b. Both were post-trained with MXFP4 weights, so the 120B model fits one 80 GB GPU and the 20B model runs within 16 GB. Each exposes low, medium and high reasoning effort and handles function calling. Thanks to that small active count, gpt-oss-120b runs at usable speeds even when most of its experts sit in system RAM, which Tutorial 5 relies on.

Nemotron

NVIDIA's Nemotron 3 Super model card describes a 120B hybrid MoE with 12B active parameters and up to one million tokens of context, released under the NVIDIA Nemotron Open Model License rather than Apache 2.0. The NVFP4 build lists a single DGX Spark as a supported minimum. Choose it for long-document and agent workloads on NVIDIA hardware where the licence suits you.

04The Hardware You Need And What It Costs

Memory decides what you can run, and memory bandwidth decides how fast it runs. A model needs room for three things: the quantized weights, the KV cache that stores attention state for every token in the context, and compute buffers. The llama.cpp guide to running gpt-oss shows the split clearly: gpt-oss-120b needs 61 GB of model data, 2.7 GB of compute buffers and about 0.3 GB of cache per 8,192 tokens, which comes to 64.9 GB at 32K context.

Context length is the setting people forget. Ollama's context-length documentation says it defaults to 4K tokens below 24 GiB of VRAM, 32K between 24 and 48 GiB, and 256K above that, and it recommends at least 64,000 tokens for coding tools. A coding agent sends its system prompt and tool schemas with every request, so a 4K window breaks it.

Prices have moved sharply this year, so treat every figure as approximate for October 2026. US figures for new and used GPUs come from BuySellRam's October GPU price floors. Indian figures come from GPUPrix listings for the RTX 5060 Ti and the RTX 5090 in India. Where no Indian listing exists, the table converts at ₹88 per dollar; Runyard's landed-cost breakdown explains why duty, 18 percent IGST and margins usually add 32 to 55 percent on top. Unified-memory machines are priced from ComputingForGeeks' Ryzen AI Max+ 395 comparison and the DGX Spark price report. Build totals add a CPU, board, RAM, PSU and SSD and are my estimates.

A used RTX 3090 remains the cheapest route to 24 GB, which is the size where Qwen3.8-27B and Gemma 4 31B become comfortable. For 70B and 120B models, a 128 GB unified-memory box trades speed for capacity: the DGX Spark's memory runs at 273 GB/s, a fraction of a desktop GPU's bandwidth, so dense 70B models generate slowly while MoE models with few active parameters stay quick.

05Five Tools That Run LLMs Locally

Each tool below loads model weights, runs them on your GPU and exposes an HTTP API. They differ in interface, API dialect and how many users they serve at once.

Ollama is the quickest start. One script installs a CLI, a background service and a model registry, and ollama run downloads and starts a model in one step. It serves an OpenAI-compatible API under /v1 and, since version 0.14, an Anthropic-compatible one, and its ollama launch command configures Claude Code and OpenCode for you. Ollama's Linux install guide covers NVIDIA and AMD ROCm setups.

LM Studio is a desktop app with a model browser, per-model settings and a headless lms command line for servers. It exposes an OpenAI-compatible API on port 1234, and LM Studio's Claude Code announcement introduced a /v1/messages endpoint in version 0.4.1, documented on its Anthropic compatibility page. It suits people who want to compare quantizations and sampling settings without editing files.

llama.cpp is the engine underneath most desktop tools, and its llama-server binary is the most configurable server here. The llama-server README lists OpenAI-compatible chat, responses and embeddings routes plus Anthropic Messages API compatibility, with a default port of 8080. It builds against CUDA, ROCm or Vulkan, and flags such as --n-cpu-moe let a 120B MoE run on a modest GPU.

vLLM is the throughput option for serving several people at once. It batches concurrent requests on the GPU, serves an OpenAI-compatible API on port 8000, and, per vLLM's Claude Code integration guide, implements the Anthropic Messages API as well. It loads safetensors checkpoints such as FP8 and NVFP4 rather than GGUF files, and it expects a card with enough VRAM for the whole model.

Jan is an open-source desktop chat app built on llama.cpp. Jan's local API server documentation describes an OpenAI-compatible server at 127.0.0.1:1337, and Linux builds come as .deb and AppImage packages. Its docs do not describe an Anthropic endpoint, so pair it with OpenCode or your own scripts rather than Claude Code.

06Local LLMs With OpenCode On Linux

OpenCode is an open-source coding agent for the terminal. It reads your repository, plans changes, edits files and runs commands, and it talks to models through the AI SDK, which lets it use any OpenAI-compatible server. OpenCode's provider documentation shows the exact block for Ollama, LM Studio and llama.cpp: a provider entry that uses the @ai-sdk/openai-compatible package, an options.baseURL pointing at the local server, and a models map whose keys must match the server's model IDs exactly, colons included.

Two settings decide whether the agent works. The first is context. OpenCode sends a long system prompt and tool definitions with each turn, so the server must allow far more than its default; set Ollama to 64K tokens and declare the same limit.context in the config so OpenCode can track how much room is left. The second is tool calling. The agent edits files and runs commands through structured tool calls, and a model that writes a tool call as plain text stalls the loop. OpenCode's docs suggest raising num_ctx first when tool calls fail, then switching to a model with stronger tool calling.

Start with models whose cards list native function calling: Qwen3.8-27B on a 24 GB card, gpt-oss-20b on a 16 GB card, and gpt-oss-120b or Mistral Small 4 on a 128 GB machine. Ollama can also write the config with ollama launch opencode --model qwen3.8, which leaves your own opencode.json untouched. Tutorial 2 builds it by hand so you can see each key.

07Local LLMs With Claude Code And Claude

There are two ways to combine Claude with a local model.

The first runs Claude Code, Anthropic's terminal agent, against a local model instead of Anthropic's API. Claude Code speaks the Anthropic Messages API, and Claude Code's environment variable reference documents ANTHROPIC_BASE_URL for redirecting it. Ollama's Claude Code integration page gives the full setup: set the base URL to http://localhost:11434, set ANTHROPIC_AUTH_TOKEN to ollama, leave ANTHROPIC_API_KEY empty, and start claude --model with a local model name. Claude Code also asks for Opus, Sonnet and Haiku tiers internally, so map all three with ANTHROPIC_DEFAULT_OPUS_MODEL, ANTHROPIC_DEFAULT_SONNET_MODEL and ANTHROPIC_DEFAULT_HAIKU_MODEL.

What works: file reads and edits, shell commands, and multi-step tasks on a repository with a model that has native tool calling and at least 64K of context. What degrades: speed, because local generation is slower than Anthropic's servers; long sessions, where small models lose track sooner; and some hosted features such as Anthropic's server-side web search. vLLM adds one rule of its own: served model names cannot contain a slash, so start it with --served-model-name.

The second pattern keeps Claude in the cloud and gives it a local model as a tool through the Model Context Protocol (MCP). Claude Code's MCP documentation shows how to register a local stdio server with claude mcp add. The server reads a file on disk, asks the local model to summarise or classify it, and returns only the result. Claude does the planning and hard reasoning, while bulk reading and private documents stay on your machine, as Figure 4 shows.

08Six Hands-On Tutorials

The tutorials build on each other, but each one runs on its own. Basic tutorials need a 16 GB GPU or better, intermediate ones a 24 GB card for the suggested model, and the advanced ones at least 16 GB of VRAM plus 64 GB of system RAM. Use the filter to show one level at a time.

# Step 1: install Ollama with the official script.
# The script installs the binary, creates an "ollama" system user
# and registers a systemd service that starts on boot.
curl -fsSL https://ollama.com/install.sh | sh

# Check that the service is active and print the installed version.
systemctl status ollama --no-pager
ollama -v

# Step 2: download Gemma 4 E4B (about 6.6 to 9.5 GB) and ask it a question.
# "ollama run" loads the model into VRAM and prints the reply.
ollama pull gemma4:e4b
ollama run gemma4:e4b "Explain what a systemd unit file is in two sentences."

# Step 3a: call Ollama's native chat API.
# "stream": false returns one JSON object instead of a token stream.
curl -s http://localhost:11434/api/chat -d '{
  "model": "gemma4:e4b",
  "messages": [{"role": "user", "content": "Name three Linux file systems."}],
  "stream": false
}'

# Step 3b: call the OpenAI-compatible endpoint that most tools use.
# Any OpenAI client library can point at http://localhost:11434/v1.
curl -s http://localhost:11434/v1/chat/completions   -H "Content-Type: application/json"   -d '{
    "model": "gemma4:e4b",
    "messages": [{"role": "user", "content": "Reply with the word ready."}]
  }'

# Confirm the model runs fully on the GPU: PROCESSOR should read "100% GPU".
ollama ps

# Step 1: set a 64K default context for every model Ollama loads.
# The drop-in file overrides the packaged unit without editing it.
sudo mkdir -p /etc/systemd/system/ollama.service.d
printf '[Service]
Environment="OLLAMA_CONTEXT_LENGTH=65536"
'   | sudo tee /etc/systemd/system/ollama.service.d/context.conf
sudo systemctl daemon-reload
sudo systemctl restart ollama

# Step 2: install OpenCode with its official script.
curl -fsSL https://opencode.ai/install | bash

# Pull a model with native tool calling.
# 24 GB GPU: Qwen3.8-27B (18 GB). 16 GB GPU: use gpt-oss:20b instead.
ollama pull qwen3.8:27b

# Step 3: create the global OpenCode config directory.
mkdir -p ~/.config/opencode

{
  "$schema": "https://opencode.ai/config.json",
  "model": "ollama/qwen3.8:27b",
  "provider": {
    "ollama": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Ollama (local)",
      "options": {
        "baseURL": "http://localhost:11434/v1"
      },
      "models": {
        "qwen3.8:27b": {
          "name": "Qwen3.8 27B (local)",
          "limit": { "context": 65536, "output": 16384 }
        },
        "gpt-oss:20b": {
          "name": "gpt-oss 20B (local)",
          "limit": { "context": 65536, "output": 16384 }
        }
      }
    }
  }
}

# Create a tiny project for the agent to work on.
mkdir -p ~/oc-demo && cd ~/oc-demo
cat > greet.py <<'EOF'
def greet(name):
    # Deliberately missing input validation for the agent to add.
    return "Hello, " + name
EOF

# Start OpenCode in this directory. Inside the TUI:
#   1. type /models and pick "Qwen3.8 27B (local)"
#   2. ask: "Add a check to greet() that raises ValueError for empty names,
#      then add a unittest file and run it."
opencode

# Step 1: install Claude Code with Anthropic's native installer.
curl -fsSL https://claude.ai/install.sh | bash

# Step 2: build a model variant whose context is fixed at 64K.
# A Modelfile layers parameters on top of an existing model.
mkdir -p ~/modelfiles && cd ~/modelfiles
cat > Modelfile.qwen64k <<'EOF'
FROM qwen3.8:27b
PARAMETER num_ctx 65536
EOF
ollama create qwen3.8-64k -f Modelfile.qwen64k

# Step 3: write a wrapper so the local settings apply only when you want them.
mkdir -p ~/.local/bin
cat > ~/.local/bin/claude-local <<'EOF'
#!/usr/bin/env bash
# claude-local: run Claude Code against the local Ollama server.
# Point Claude Code at Ollama's Anthropic-compatible API.
export ANTHROPIC_BASE_URL="http://localhost:11434"
# Ollama ignores the token's value, but Claude Code needs one set.
export ANTHROPIC_AUTH_TOKEN="ollama"
# An empty API key stops Claude Code from preferring a real key.
export ANTHROPIC_API_KEY=""
# Map Claude Code's internal Opus, Sonnet and Haiku tiers to the local model.
export ANTHROPIC_DEFAULT_OPUS_MODEL="qwen3.8-64k"
export ANTHROPIC_DEFAULT_SONNET_MODEL="qwen3.8-64k"
export ANTHROPIC_DEFAULT_HAIKU_MODEL="qwen3.8-64k"
exec claude --model qwen3.8-64k "$@"
EOF
chmod +x ~/.local/bin/claude-local

# Create a project with a failing test for the agent to fix.
mkdir -p ~/cc-local && cd ~/cc-local
cat > stats.py <<'EOF'
def mean(values):
    # Bug: the "+ 1" makes every result wrong.
    return sum(values) / len(values) + 1
EOF
cat > test_stats.py <<'EOF'
import unittest
from stats import mean

class MeanTest(unittest.TestCase):
    def test_mean(self):
        self.assertEqual(mean([2, 4, 6]), 4)

if __name__ == "__main__":
    unittest.main()
EOF

# Start Claude Code on the local model, then ask:
#   "Run python3 -m unittest, fix the bug in stats.py, and run the tests again."
claude-local

# Isolate the dependencies in a virtual environment.
python3 -m venv ~/llm-venv
source ~/llm-venv/bin/activate
pip install --upgrade openai
ollama pull gpt-oss:20b

"""Stream from a local model, measure speed, then run a tool call."""
import json
import shutil
import time

from openai import OpenAI

# Point the OpenAI client at the local server. Ollama ignores the key.
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
MODEL = "gpt-oss:20b"


def stream_and_measure(prompt: str) -> None:
    """Stream a reply and report time to first token and tokens per second."""
    start = time.perf_counter()
    first_token_at = None
    chunks = 0
    usage_tokens = None
    stream = client.chat.completions.create(
        model=MODEL,
        messages=[{"role": "user", "content": prompt}],
        stream=True,
        stream_options={"include_usage": True},  # final chunk carries token counts
    )
    for chunk in stream:
        if chunk.usage is not None:
            usage_tokens = chunk.usage.completion_tokens
        if chunk.choices and chunk.choices[0].delta.content:
            if first_token_at is None:
                first_token_at = time.perf_counter()
            chunks += 1
            print(chunk.choices[0].delta.content, end="", flush=True)
    end = time.perf_counter()
    # Fall back to counting streamed chunks if the server sent no usage data.
    tokens = usage_tokens or chunks
    gen_time = end - (first_token_at or start)
    print(f"

Time to first token: {(first_token_at or end) - start:.2f} s")
    print(f"Generated {tokens} tokens at {tokens / max(gen_time, 1e-6):.1f} tokens/s")


def disk_free_gb(path: str) -> dict:
    """The real function the model can call: free space at a path, in GB."""
    usage = shutil.disk_usage(path)
    return {"path": path, "free_gb": round(usage.free / 1e9, 1)}


TOOLS = [{
    "type": "function",
    "function": {
        "name": "disk_free_gb",
        "description": "Return the free disk space in gigabytes for a filesystem path.",
        "parameters": {
            "type": "object",
            "properties": {"path": {"type": "string", "description": "A path such as /"}},
            "required": ["path"],
        },
    },
}]


def tool_round_trip(question: str) -> None:
    """Ask a question, execute any tool the model requests, then get the answer."""
    messages = [{"role": "user", "content": question}]
    first = client.chat.completions.create(model=MODEL, messages=messages, tools=TOOLS)
    reply = first.choices[0].message
    if not reply.tool_calls:
        print("Model answered without a tool:", reply.content)
        return
    # Keep the assistant turn, then append one tool result per call.
    messages.append(reply.model_dump(exclude_none=True))
    for call in reply.tool_calls:
        args = json.loads(call.function.arguments or "{}")
        result = disk_free_gb(**args)
        print(f"Tool call: {call.function.name}({args}) -> {result}")
        messages.append({"role": "tool", "tool_call_id": call.id, "content": json.dumps(result)})
    final = client.chat.completions.create(model=MODEL, messages=messages, tools=TOOLS)
    print("Answer:", final.choices[0].message.content)


if __name__ == "__main__":
    stream_and_measure("Explain the difference between RAM and VRAM in 80 words.")
    tool_round_trip("How much free disk space is on / right now?")

# Step 1: build tools and libcurl (needed for -hf downloads).
# Fedora: sudo dnf install gcc-c++ cmake git libcurl-devel
# Arch:   sudo pacman -S base-devel cmake git curl
sudo apt update && sudo apt install -y build-essential cmake git libcurl4-openssl-dev

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

# NVIDIA build. For AMD use -DGGML_HIP=ON (ROCm) or -DGGML_VULKAN=ON instead.
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j "$(nproc)"

# Install binaries and shared libraries under /usr/local, then refresh the loader cache.
sudo cmake --install build --prefix /usr/local
sudo ldconfig

# Step 2: create a service account and a model cache it owns.
sudo useradd -r -m -d /var/lib/llama -s /usr/sbin/nologin llama

# First run as that user downloads about 61 GB into LLAMA_CACHE, then serves.
# --n-cpu-moe 32 keeps 32 MoE layers in system RAM; lower it if VRAM allows.
sudo -u llama env LLAMA_CACHE=/var/lib/llama/cache   llama-server -hf ggml-org/gpt-oss-120b-GGUF   --ctx-size 32768 --jinja -ub 4096 -b 4096   --n-cpu-moe 32 --host 127.0.0.1 --port 8080 --alias gpt-oss-120b

# Quick test against the OpenAI-compatible route.
curl -s http://127.0.0.1:8080/v1/chat/completions -H "Content-Type: application/json"   -d '{"model":"gpt-oss-120b","messages":[{"role":"user","content":"Say hello in Tamil."}]}'

# Runs llama-server for gpt-oss-120b at boot and restarts it on failure.
[Unit]
Description=llama.cpp server for gpt-oss-120b
After=network-online.target
Wants=network-online.target

[Service]
User=llama
Group=llama
# Reuse the cache filled during the first run, so nothing downloads again.
Environment=LLAMA_CACHE=/var/lib/llama/cache
ExecStart=/usr/local/bin/llama-server -hf ggml-org/gpt-oss-120b-GGUF --ctx-size 32768 --jinja -ub 4096 -b 4096 --n-cpu-moe 32 --host 127.0.0.1 --port 8080 --alias gpt-oss-120b
Restart=on-failure
RestartSec=5
# Lets the process lock model pages in RAM if you later add --mlock.
LimitMEMLOCK=infinity

[Install]
WantedBy=multi-user.target

# Step 3: enable the service and watch it load the model.
sudo systemctl daemon-reload
sudo systemctl enable --now llama-gptoss
journalctl -u llama-gptoss -f   # press Ctrl+C once it reports it is listening

# Benchmark prompt processing (pp) and generation (tg) with the same offload.
# Stop the service first so the benchmark has the GPU to itself.
sudo systemctl stop llama-gptoss
MODEL=$(sudo find /var/lib/llama/cache -name '*120b*00001-of-*.gguf' | head -n1)
sudo -u llama llama-bench -m "$MODEL" -fa 1 -p 2048 -n 128 --n-cpu-moe 32
sudo systemctl start llama-gptoss

"""MCP server that exposes a local Ollama model to Claude as two tools.

Raw file contents are processed by the local model. Only the summary
or label that each tool returns is sent back to Claude.
"""
import json
import os
import urllib.request
from pathlib import Path

from mcp.server import MCPServer

# Local OpenAI-compatible endpoint and model; override with environment variables.
LOCAL_URL = os.environ.get("LOCAL_LLM_URL", "http://localhost:11434/v1/chat/completions")
LOCAL_MODEL = os.environ.get("LOCAL_LLM_MODEL", "qwen3.8:27b")
MAX_CHARS = 60_000  # cap input so the request fits the local context window

mcp = MCPServer("local-llm")


def ask_local(system: str, user: str) -> str:
    """Send one chat request to the local model using only the standard library."""
    payload = json.dumps({
        "model": LOCAL_MODEL,
        "messages": [
            {"role": "system", "content": system},
            {"role": "user", "content": user},
        ],
        "stream": False,
    }).encode()
    request = urllib.request.Request(
        LOCAL_URL, data=payload, headers={"Content-Type": "application/json"}
    )
    with urllib.request.urlopen(request, timeout=600) as response:
        data = json.load(response)
    return data["choices"][0]["message"]["content"]


@mcp.tool()
def summarize_local_file(path: str, focus: str = "key points and decisions") -> str:
    """Summarise a local text file with the on-device model and return only the summary."""
    text = Path(path).expanduser().read_text(errors="replace")[:MAX_CHARS]
    return ask_local(
        "You summarise documents accurately in under 200 words.",
        f"Focus on: {focus}

---
{text}",
    )


@mcp.tool()
def classify_text(text: str, labels: list[str]) -> str:
    """Classify text into exactly one of the given labels using the on-device model."""
    answer = ask_local(
        "Reply with exactly one label from the list and nothing else.",
        f"Labels: {', '.join(labels)}

Text:
{text[:MAX_CHARS]}",
    )
    return answer.strip()


if __name__ == "__main__":
    mcp.run()  # with no arguments the server speaks MCP over stdio

# Register the server with Claude Code. Everything after "--" is the launch command.
# uv resolves the MCP SDK on the fly, so no virtual environment is needed.
claude mcp add --env LOCAL_LLM_MODEL=qwen3.8:27b --transport stdio local-llm   -- uv run --with "mcp[cli]" mcp run "$HOME/mcp/local_llm_mcp.py"

# Confirm it connected; the status column should show a check mark.
claude mcp list

# Start a normal (cloud) Claude Code session and try:
#   "Use summarize_local_file on ~/notes/contract-draft.txt and list any deadlines."
claude

# Three model groups behind one OpenAI-compatible endpoint on port 4000.
model_list:
  - model_name: local-private        # private data: local only, never falls back
    litellm_params:
      model: ollama_chat/qwen3.8:27b
      api_base: http://localhost:11434
  - model_name: local-fast           # routine work: local first
    litellm_params:
      model: ollama_chat/qwen3.8:27b
      api_base: http://localhost:11434
  - model_name: claude-hard          # hard reasoning: Claude Sonnet 5.5
    litellm_params:
      model: anthropic/claude-sonnet-5-5
      api_key: os.environ/ANTHROPIC_API_KEY

litellm_settings:
  num_retries: 2
  # If the local server errors, retry routine requests on Claude.
  fallbacks: [{"local-fast": ["claude-hard"]}]

"""Pick a model group per task and send it through the LiteLLM proxy."""
from openai import OpenAI

# The proxy has no master key configured, so any placeholder key works locally.
client = OpenAI(base_url="http://localhost:4000/v1", api_key="sk-local-proxy")

HARD_HINTS = ("architecture", "prove", "refactor the whole", "security review", "design")


def choose_model(task: str, private: bool) -> str:
    """Private data stays local; hard or very long tasks go to Claude."""
    if private:
        return "local-private"
    if len(task) > 8000 or any(hint in task.lower() for hint in HARD_HINTS):
        return "claude-hard"
    return "local-fast"


def run(task: str, private: bool = False) -> str:
    model = choose_model(task, private)
    reply = client.chat.completions.create(
        model=model, messages=[{"role": "user", "content": task}]
    )
    return f"[{model}] {reply.choices[0].message.content}"


if __name__ == "__main__":
    print(run("Write a regex that matches Indian PIN codes."))
    print(run("Summarise this salary sheet: ...", private=True))
    print(run("Do a security review of this login design: ..."))

# Install the proxy and the OpenAI SDK into the venv from Tutorial 4.
source ~/llm-venv/bin/activate
pip install 'litellm[proxy]' openai

# Keep the real key in the environment, never in the config file.
export ANTHROPIC_API_KEY="YOUR_ANTHROPIC_API_KEY"

# Start the proxy in one terminal...
litellm --config ~/litellm/config.yaml --port 4000

# ...and run the router in another.
python ~/litellm/router.py

09Conclusion: Choosing Your Setup

Start from the hardware you have. A 16 GB card runs the 8B and 14B models, gpt-oss-20b and, with 64 to 96 GB of system RAM, gpt-oss-120b through expert offload. A 24 GB card is the point where Qwen3.8-27B makes local coding agents useful. For 70B dense models or several 120B MoE models, a 128 GB unified-memory machine gives capacity, while a GPU gives speed.

Before you buy, price your workload. If your team spends hundreds of dollars a month on Sonnet or Opus tokens for work a local model can handle, the break-even calculator will show a payback measured in months. If you mostly send short prompts that Haiku 5.5 handles for cents, local hardware pays off only through privacy, offline access or the freedom to run unlimited batch jobs.

Then wire it in gradually. Begin with Ollama and OpenCode, move to Claude Code on a local backend for private repositories, and add the MCP server so cloud Claude can hand bulk reading to your GPU. Keep Claude for the work where its lead matters, and let the local model take the rest.

10References

1. Ollama blog: Claude Code with Anthropic API compatibility
2. Ollama docs: Claude Code integration and environment variables
3. Ollama docs: Linux installation, systemd service and ROCm
4. Ollama docs: context length defaults and settings
5. Ollama library: qwen3.8 model tags and sizes
6. Ollama library: gemma4 model tags and sizes
7. LM Studio docs: Anthropic-compatible Messages endpoint & LM Studio blog: using LM Studio models in Claude Code
8. llama.cpp: llama-server README and API compatibility
9. llama.cpp discussion: guide to running gpt-oss with llama.cpp
10. llama.cpp docs: building with CUDA, HIP and Vulkan
11. vLLM docs: using Claude Code with a vLLM server
12. Jan docs: local OpenAI-compatible API server
13. OpenCode docs: providers, local models and custom baseURL
14. Claude Code docs: environment variable reference
15. Claude Code docs: connecting tools through MCP
16. MCP Python SDK: connecting a server to a real host
17. LiteLLM docs: proxy fallbacks and reliability settings
18. Claude Platform docs: API pricing for Opus, Sonnet and Haiku 5.5
19. Hugging Face: Qwen3.8-27B model card
20. Google blog: Gemma 4 announcement and model sizes
21. Hugging Face: Ministral 3 8B Instruct model card
22. Hugging Face: Mistral Small 4 119B model card
23. Hugging Face: Llama 3.3 70B Instruct model card
24. Hugging Face: gpt-oss-120b model card
25. Hugging Face: Nemotron 3 Super 120B NVFP4 model card
26. Pinggy: best open-source LLMs for coding, October 2026
27. BuySellRam: NVIDIA consumer GPU price update, October 2026
28. GPUPrix: RTX 5060 Ti 16GB price in India & RTX 5090 price in India
29. ComputingForGeeks: Ryzen AI Max+ 395 mini PC prices & Crypto Briefing: DGX Spark 64GB launch and pricing
30. US EIA: Short-Term Energy Outlook, residential electricity prices & Runyard: Indian GPU landed costs