I've tried running local models for coding before. It's always the same story: you get excited, spend an hour downloading something, and then it spits out code that's slightly worse than asking a rubber duck. So I've mostly stuck to API calls.
Qwen 3.6 27B changed that for me. Genuinely.
What makes it different
Alibaba's Qwen 3.6 comes in two flavours: a 35B mixture-of-experts model (Qwen 3.6 35B A3B) and a denser 27B. The 27B is slower, but meaningfully smarter for coding tasks. The consistent take from people benchmarking it is that it "punches above its weight" — which is code for: it behaves like a model 2-3x its size.
What that means practically: you can give it a prompt like "create a hexagonal minesweeper using pnpm" and it builds a proper Node package that works on the first go. That's not impressive compared to GPT-4o, but it's miles ahead of what local inference looked like even six months ago.
The other factor is multi-token prediction (MTP) support. Running the MTP-enabled GGUF means the model speculates ahead on token generation — you get noticeably faster output without sacrificing quality.
Setting it up with llama.cpp
You need llama.cpp. Not Ollama — the article I'm drawing from recommends skipping Ollama on ethical grounds, and even setting that aside, llama.cpp gives you more direct control.
Grab the Q8_0 quantised version from Hugging Face (8-bit, barely any quality loss, roughly half the file size of BF16):
llama-server -hf unsloth/Qwen3.6-27B-MTP-GGUF:Q8_0 \
--spec-type draft-mtp \
-ngl 999 \
-fa on \
-c 65536 \
--port 8080
The flags that matter:
--spec-type draft-mtp— enables multi-token prediction for speed-ngl 999— pushes all layers to GPU (if you have one; on Apple Silicon this maps to the unified memory)-fa on— flash attention-c 65536— sets context to 64k (native is 256k, but that chews more RAM)
Once it's running, http://127.0.0.1:8080 gives you a chat UI. The same server exposes an OpenAI-compatible API at /v1, which means you can point any tool that speaks the OpenAI API spec at it.
Wiring it into your coding workflow
The llama.cpp server speaks the OpenAI API format. That means you can drop it into tools like OpenCode with a config tweak:
// ~/.config/opencode/opencode.jsonc
{
"$schema": "https://opencode.ai/config.json",
"provider": {
"llama": {
"name": "llama.cpp (local)",
"npm": "@ai-sdk/openai-compatible",
"options": {
"baseURL": "http://127.0.0.1:8080/v1",
"apiKey": "local"
},
"models": {
"qwen3.6-27b": {
"name": "Qwen3.6-27B Q8 +MTP"
}
}
}
},
"model": "llama/qwen3.6-27b"
}
Same pattern works anywhere you can configure an OpenAI base URL — Cursor, Continue.dev, or a custom script using the openai Node package pointed at http://localhost:8080/v1.
Hardware reality check: you need enough RAM to hold the model. The Q8_0 of 27B weighs about 29GB. A MacBook Pro M3/M4/M5 with 36GB+ unified memory runs it comfortably. An Nvidia RTX 4090 with 24GB VRAM can load it with a bit of layer offloading. Anything less and you're looking at the smaller 35B A3B MoE variant, which is faster but less precise on hard coding tasks.
What I'd build with this
Private code analysis pipeline. Throw your internal codebase at a local Qwen instance and ask it to find security issues, suggest refactors, or explain legacy modules. No data leaves your machine, no API costs, no terms-of-service gymnastics.
Local RAG over docs. Embed your team's internal documentation, Confluence exports, or Notion dumps into a vector store (pgvector on a local Postgres instance works fine), and wire up Qwen as the generation layer. You get a "chat with our docs" tool that answers questions about your actual internal systems — not just public Stack Overflow answers.
Async code review bot. Run a local agent that polls a GitHub PR feed, pulls diffs, and posts comments via the GitHub API. You'd want the smaller, faster A3B model for throughput, but for a team of 3-5 people this could replace a chunk of manual review time for things like "did you forget to add RLS?" or "is this SQL injectable?"
My take
The cost angle is real but not the main story. Yes, running inference locally means no per-token charges, which matters if you're prototyping something that hammers an LLM 500 times to test a prompt. But the more interesting thing is latency and iteration speed. There's no cold start, no rate limit, no context window anxiety the way you get with hosted tiers. You can fire off 10 exploratory prompts in 30 seconds and it doesn't feel like you're burning money.
The 27B model isn't going to replace Claude Sonnet or GPT-4o for hard multi-step reasoning. But for the majority of coding tasks — boilerplate generation, explaining a function, writing a test, drafting a migration — it's there. And "there" while running on your own machine is a meaningfully different thing.
Worth downloading today.