Why Fine-Tune for Code?

Generic code models (GPT-4, Claude, Gemini) are excellent at general programming tasks. But they don't know your codebase — your naming conventions, architecture patterns, internal APIs, or preferred libraries. Fine-tuning a local model on your code makes completions feel like they were written by someone on your team.

Benefits of fine-tuned local models:

  • Relevance: Completions match your project style and patterns
  • Privacy: Code never leaves your machine
  • Cost: Zero API costs after training
  • Speed: Local inference is often faster than cloud APIs
  • Offline: Works without internet

For a general LLM overview, see our LLM fine-tuning guide.

Choosing a Base Model

  • DeepSeek Coder 33B: Best overall quality. Competes with GPT-4 for coding. Needs 24GB VRAM (quantized).
  • CodeLlama 13B: Best balance of speed and quality. Runs on 16GB VRAM. Meta's code-specialized Llama.
  • StarCoder2 15B: Trained on 600+ languages. Excellent for multi-language teams. BigCode open-source project.
  • DeepSeek Coder 6.7B: Best small model. Runs fast on 8GB VRAM. Good for laptops.

Preparing Your Training Dataset

Quality matters more than quantity. Here's how to prepare your dataset:

1. Extract Examples from Your Codebase

# Extract function-level examples
import ast, json, pathlib examples = []
for py_file in pathlib.Path("src").rglob("*.py"): source = py_file.read_text() tree = ast.parse(source) for node in ast.walk(tree): if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)): func_source = ast.get_source_segment(source, node) if func_source and len(func_source) > 50: examples.append({ "instruction": f"Write a Python function: {node.name}", "output": func_source }) with open("dataset.jsonl", "w") as f: for ex in examples: f.write(json.dumps(ex) + "\n")

2. Format for Training

# Alpaca-style format (most compatible)
{ "instruction": "Implement a retry decorator with exponential backoff", "input": "", "output": "def retry(max_retries=3, backoff_factor=2):..."
}

Fine-Tuning with QLoRA

# Install dependencies
pip install transformers peft bitsandbytes datasets accelerate trl # Training script (simplified)
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model
from trl import SFTTrainer # Load model in 4-bit quantization
bnb_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16,
) model = AutoModelForCausalLM.from_pretrained( "deepseek-ai/deepseek-coder-6.7b-instruct", quantization_config=bnb_config, device_map="auto",
) # LoRA config — only train small adapter layers
lora_config = LoraConfig( r=16, lora_alpha=32, target_modules=["q_proj", "v_proj"], lora_dropout=0.05, task_type="CAUSAL_LM",
) model = get_peft_model(model, lora_config) # Train
trainer = SFTTrainer( model=model, train_dataset=dataset, max_seq_length=2048, num_train_epochs=3, per_device_train_batch_size=4,
)
trainer.train()
model.save_pretrained("my-code-model-lora")

Deploying with Ollama

# Merge LoRA weights and convert to GGUF
python merge_lora.py --base deepseek-coder-6.7b --lora my-code-model-lora --output merged/ # Convert to GGUF format
python llama.cpp/convert.py merged/ --outtype q4_K_M --outfile my-code-model.gguf # Create Ollama Modelfile
cat > Modelfile << EOF
FROM./my-code-model.gguf
PARAMETER temperature 0.2
PARAMETER top_p 0.9
SYSTEM "You are a code assistant trained on our codebase."
EOF # Import into Ollama
ollama create my-code-model -f Modelfile
ollama run my-code-model

Tips for Better Results

  • Clean your dataset: Remove auto-generated code, tests, and boilerplate
  • Include docstrings: Models learn to generate documentation too
  • Version your models: Track training data, hyperparameters, and eval results
  • Evaluate regularly: Use a held-out test set of code completions to measure quality
  • Combine with RAG: Fine-tune for style, use RAG for up-to-date API documentation

Conclusion

Fine-tuning local LLMs for code is one of the most impactful skills for developers in 2026. With QLoRA, you can train on consumer hardware, deploy via Ollama, and get completions that understand your codebase — all at zero ongoing cost. Start with a small model (6.7B), prove the value with your team, then scale to larger models.

Get started with our local LLM setup guide and try running models with free AI coding agents.