You have a laptop with 16GB of RAM. You want to run an AI model locally. You search online and find dozens of options: Llama, Mistral, Phi, Qwen, DeepSeek, CodeGemma, Gemma. Each has multiple sizes, quantization levels, and specializations.
Which one should you choose?
The answer depends on six factors:
- RAM — How much memory is available?
- GPU — Do you have a dedicated graphics card or Apple Silicon?
- Task — What are you trying to do?
- Language — What programming or natural language do you need?
- Context — How long are your inputs?
- Speed — How fast do you need responses?
This article provides a practical decision framework. We will not declare a universal winner. Instead, we will help you find the right model for your specific situation.
Decision Tree: How to Choose
The decision process follows three main branches:
- Check your RAM — This determines which model sizes are feasible
- Identify your task — Different tasks benefit from different model specializations
- Assess GPU availability — This affects speed and quantization choices
RAM Tiers and Model Sizes
Your available RAM is the primary constraint. Here is what each tier can handle:
| RAM Tier | Model Size | Quantization | Example Models | Best For |
|---|---|---|---|---|
| 8GB | 3B parameters | Q4_K_M | Phi-4 Mini, Llama 3.2 3B, Qwen 2.5 3B | Quick chat, simple tasks |
| 16GB | 7-8B parameters | Q4_K_M to Q5_K_M | Mistral 7B, Llama 3.1 8B, Qwen 2.5 7B | Most tasks, coding, chat |
| 32GB | 13-30B parameters | Q4_K_M to Q6_K | Llama 3.1 13B, Qwen 2.5 14B, Mixtral 8x7B | Complex reasoning, RAG |
| 64GB+ | 30-70B parameters | Q4_K_M to Q8_0 | Llama 3.1 70B, Qwen 2.5 72B | Maximum quality, research |
Task-Specific Recommendations
💻 Coding
For code generation, debugging, and programming assistance:
| RAM | Recommended Model | Why |
|---|---|---|
| 8GB | Phi-4 Mini (3.8B) | Fast, good code completion |
| 16GB | DeepSeek Coder V2 Lite (16B) | Specialized for code, strong performance |
| 16GB | CodeGemma 7B | Google's code model, good at Python/JS |
| 32GB | DeepSeek Coder V2 (236B MoE) | Advanced coding, multi-language |
💬 Chat and Conversation
For general conversation, Q&A, and interactive use:
| RAM | Recommended Model | Why |
|---|---|---|
| 8GB | Llama 3.2 3B | Versatile, good instruction following |
| 16GB | Llama 3.1 8B | Excellent balance of quality and speed |
| 16GB | Phi-4 (14B) | Strong reasoning, Microsoft research |
| 32GB | Qwen 2.5 14B | Multilingual, strong performance |
📝 Summarization
For condensing documents, articles, and long texts:
| RAM | Recommended Model | Why |
|---|---|---|
| 8GB | Phi-4 Mini (3.8B) | Fast, handles short documents |
| 16GB | Mistral 7B | Efficient, good at extraction |
| 16GB | Qwen 2.5 7B | Strong summarization capabilities |
| 32GB | Llama 3.1 13B | Better coherence for long documents |
🔍 RAG (Retrieval-Augmented Generation)
For question-answering over your own documents:
| RAM | Recommended Model | Why |
|---|---|---|
| 8GB | Phi-4 Mini (3.8B) | Fast, good for simple RAG |
| 16GB | Llama 3.1 8B | Good context handling, reliable |
| 16GB | Mistral 7B | Efficient, good instruction following |
| 32GB | Llama 3.1 13B | Longer context, better reasoning |
GPU vs CPU: What Changes?
If you have a dedicated GPU (NVIDIA with 8GB+ VRAM) or Apple Silicon, you can run larger models faster. Here is how GPU availability affects your choices:
| Hardware | Speed | Quantization | Best Approach |
|---|---|---|---|
| CPU only | 5-15 tok/s | Q4_K_M (aggressive) | Prioritize small models |
| NVIDIA 8GB VRAM | 30-60 tok/s | Q5_K_M to Q6_K | Fit model in VRAM |
| NVIDIA 12GB+ VRAM | 40-80 tok/s | Q6_K to Q8_0 | Higher quality quantization |
| Apple Silicon (16GB) | 30-50 tok/s | Q4_K_M to Q5_K_M | Unified memory advantage |
| Apple Silicon (32GB+) | 40-80 tok/s | Q5_K_M to Q8_0 | Run larger models |
Language Support
If you need multilingual support or specific programming languages:
| Need | Recommended Models | Notes |
|---|---|---|
| English only | Llama 3.1, Phi-4, Mistral | Most models excel at English |
| Multilingual | Qwen 2.5, Llama 3.1 | Qwen has strong CJK support |
| Python coding | DeepSeek Coder, CodeGemma | Specialized code models |
| Multiple languages | DeepSeek Coder V2 | Supports 338 languages |
Context Length Considerations
Context length determines how much text the model can process at once. Longer context requires more memory:
| Context Length | Memory Overhead | Use Case | Recommended |
|---|---|---|---|
| 4K tokens | Low | Quick chat, simple tasks | Any model |
| 8K tokens | Medium | Most conversations, code | Llama 3.1, Mistral |
| 32K tokens | High | Long documents, RAG | Llama 3.1, Qwen 2.5 |
| 128K tokens | Very High | Book-length analysis | Requires 32GB+ RAM |
Speed vs Quality Tradeoffs
Faster responses often mean lower quality. Here is how to balance speed and quality:
| Priority | Approach | Tradeoff |
|---|---|---|
| Maximum speed | Small model (3B), Q4 quantization | Lower quality, limited capabilities |
| Balanced | Medium model (7-8B), Q4-Q5 quantization | Good speed, good quality |
| Maximum quality | Large model (13B+), Q6-Q8 quantization | Slower, requires more RAM |
Practical Selection Guide
Step 1: Check Your Hardware
Run these commands to understand your system:
# Check available RAM
# Windows
systeminfo | findstr "Memory"
# macOS
sysctl hw.memsize | awk '{print $2/1024/1024/1024 " GB"}'
# Linux
free -h
# Check GPU (if NVIDIA)
nvidia-smi
# Check Apple Silicon
system_profiler SPDisplaysDataTypeStep 2: Start with a Small Model
Install Ollama and test a small model first:
# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh
# Pull a small model
ollama pull phi4-mini
# Test it
ollama run phi4-mini "Hello, what can you do?"Step 3: Benchmark Your Hardware
Measure tokens per second to understand your actual performance:
# Run a benchmark
ollama run llama3.1:8b-q4_K_M "Write a short story about a robot."
# Check the speed in the output
# Look for tokens/second metricStep 4: Scale Up Gradually
If your hardware can handle it, try larger models:
# Try 7B model
ollama pull mistral
ollama run mistral
# Try 13B model (if you have 32GB RAM)
ollama pull llama3.1:13b
ollama run llama3.1:13bComplete Model Selection Matrix
| Model | Size | RAM Needed | Best Task | Speed | Quality |
|---|---|---|---|---|---|
| Phi-4 Mini | 3.8B | 6-8GB | Chat, quick tasks | ⭐⭐⭐⭐⭐ | ⭐⭐⭐ |
| Llama 3.2 3B | 3B | 6-8GB | General purpose | ⭐⭐⭐⭐⭐ | ⭐⭐⭐ |
| Qwen 2.5 3B | 3B | 6-8GB | Multilingual | ⭐⭐⭐⭐⭐ | ⭐⭐⭐ |
| Mistral 7B | 7B | 10-12GB | Chat, summarization | ⭐⭐⭐⭐ | ⭐⭐⭐⭐ |
| Llama 3.1 8B | 8B | 12-14GB | General, RAG | ⭐⭐⭐⭐ | ⭐⭐⭐⭐ |
| Qwen 2.5 7B | 7B | 10-12GB | Multilingual, coding | ⭐⭐⭐⭐ | ⭐⭐⭐⭐ |
| CodeGemma 7B | 7B | 10-12GB | Python, JavaScript | ⭐⭐⭐⭐ | ⭐⭐⭐⭐ |
| Phi-4 | 14B | 18-22GB | Reasoning, complex tasks | ⭐⭐⭐ | ⭐⭐⭐⭐⭐ |
| Llama 3.1 13B | 13B | 18-22GB | Long context, RAG | ⭐⭐⭐ | ⭐⭐⭐⭐⭐ |
| Qwen 2.5 14B | 14B | 18-22GB | Multilingual, complex | ⭐⭐⭐ | ⭐⭐⭐⭐⭐ |
How to Test Models
Before committing to a model, test it with your actual workload:
# Create a test script
cat > test_model.sh << 'EOF'
#!/bin/bash
MODEL=$1
PROMPT=$2
echo "Testing $MODEL..."
echo "Prompt: $PROMPT"
echo "---"
time ollama run "$MODEL" "$PROMPT" --verbose
EOF
chmod +x test_model.sh
# Test coding
./test_model.sh phi4-mini "Write a Python function to merge two sorted lists"
# Test chat
./test_model.sh llama3.1:8b "Explain quantum computing in simple terms"
# Test summarization
./test_model.sh mistral "Summarize: [paste your text here]"Common Mistakes to Avoid
- Starting with the largest model — Begin small, scale up based on actual needs
- Ignoring quantization — Q4 is often the sweet spot for most use cases
- Not testing with real tasks — Benchmark with your actual workload
- Assuming bigger is better — A 7B model may be perfect for your needs
- Forgetting about context length — Long context requires more memory
- Not considering speed — A slow model may not be practical for interactive use
Model Selection Checklist
| Check | Question | Action |
|---|---|---|
| ☑️ | How much RAM do I have? | Check system memory |
| ☑️ | Do I have a GPU? | Check for NVIDIA/Apple Silicon |
| ☑️ | What is my primary task? | Choose model specialization |
| ☑️ | Do I need multilingual support? | Consider Qwen or Llama 3.1 |
| ☑️ | How much context do I need? | Factor in KV cache memory |
| ☑️ | What speed do I need? | Balance size vs speed |
| ☑️ | Have I tested the model? | Run benchmarks with real tasks |
Try It Yourself
Ready to experiment? Start with these BestWordz resources:
🔧 Local AI Tools
Compare Ollama, llama.cpp, and LM Studio for running local models
Read the Comparison →Conclusion
Choosing a local AI model is not about finding the "best" model. It is about finding the right model for your specific situation:
- 8GB RAM? Start with Phi-4 Mini or Llama 3.2 3B
- 16GB RAM? Mistral 7B or Llama 3.1 8B are excellent choices
- 32GB+ RAM? Explore Llama 3.1 13B or Qwen 2.5 14B
- Coding? DeepSeek Coder or CodeGemma
- Multilingual? Qwen 2.5 series
- RAG? Llama 3.1 with longer context
The best approach is to start small, test with your actual workload, and scale up only when needed. Your hardware, task, and requirements are unique—your model choice should be too.