Run LLMs Locally with Ollama: Complete Guide

Running large language models locally gives you full control over your data, eliminates API costs, and lets you experiment without rate limits. Ollama makes this surprisingly simple regardless of your operating system.

This guide walks you through installing Ollama, pulling models, customizing them, and integrating local LLMs into real workflows.

Why Run LLMs Locally?

Before diving into setup, here is why local inference matters:

• Privacy: Your prompts and data never leave your machine
• Cost: Zero API fees after the initial download
• Speed: No network latency for short prompts on capable hardware
• Offline access: Works without internet once models are downloaded
• Customization: Create custom model variants with specific system prompts and parameters

Step 1: Install Ollama

macOS

# Download and install from the official site, or use Homebrew
brew install ollama

Linux

curl -fsSL https://ollama.com/install.sh | sh

Windows

Download the installer from ollama.com and run it. Ollama runs as a background service after installation.

Verify the installation:

ollama --version

You should see the version number printed. Ollama runs a local server on port 11434 by default.

Step 2: Pull Your First Model

Ollama hosts a library of pre-quantized models ready to run. Start with a model that fits your hardware:

# Good starting point for most machines (4.7GB)
ollama pull llama3.1:8b
 
# Smaller option for limited RAM (2GB)
ollama pull phi3:mini
 
# Larger model for better quality (requires 16GB+ RAM)
ollama pull llama3.1:70b

Hardware Requirements

If you have an Apple Silicon Mac with unified memory, you can run larger models than the VRAM column suggests since system RAM is shared with the GPU.

Step 3: Run and Chat

Start an interactive session:

ollama run llama3.1:8b

This drops you into a chat interface. Type your prompt and press Enter. Use /bye to exit.

For a single non-interactive query:

ollama run llama3.1:8b "Explain the difference between TCP and UDP in two sentences"

Step 4: Use the REST API

Ollama exposes a local API compatible with the OpenAI chat completions format. This is where it gets powerful for developers:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1:8b",
  "prompt": "Write a Python function to merge two sorted lists",
  "stream": false
}'

For chat-style conversations:

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.1:8b",
  "messages": [
    {"role": "system", "content": "You are a senior Python developer."},
    {"role": "user", "content": "Review this code for bugs: def add(a, b): return a - b"}
  ],
  "stream": false
}'

Python Integration

import requests
 
def ask_ollama(prompt, model="llama3.1:8b"):
    response = requests.post(
        "http://localhost:11434/api/generate",
        json={"model": model, "prompt": prompt, "stream": False}
    )
    return response.json()["response"]
 
# Example usage
result = ask_ollama("Generate a SQL query to find duplicate emails in a users table")
print(result)

Using the OpenAI-Compatible Endpoint

Many tools support OpenAI's API format. Ollama provides a compatible endpoint:

from openai import OpenAI
 
client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama"  # any string works
)
 
response = client.chat.completions.create(
    model="llama3.1:8b",
    messages=[{"role": "user", "content": "Hello, how are you?"}]
)
print(response.choices[0].message.content)

This means any tool that supports custom OpenAI-compatible endpoints can use your local Ollama models with zero code changes.

Step 5: Create Custom Models

Modelfiles let you create specialized variants with custom system prompts, temperature settings, and more:

# Save as Modelfile
FROM llama3.1:8b
 
SYSTEM """You are a senior software engineer specializing in code review. 
Always point out potential bugs, security issues, and suggest improvements. 
Be concise and use bullet points."""
 
PARAMETER temperature 0.3
PARAMETER num_ctx 4096

Build and run your custom model:

ollama create code-reviewer -f Modelfile
ollama run code-reviewer

Now you have a dedicated code review assistant that runs entirely on your machine.

Step 6: Manage Models

Essential commands for model management:

# List downloaded models
ollama list
 
# Show model details
ollama show llama3.1:8b
 
# Remove a model to free disk space
ollama rm phi3:mini
 
# Copy a model (useful before customizing)
ollama cp llama3.1:8b my-llama

Practical Tips

Choose the right quantization: Models come in different quantization levels (Q4, Q5, Q8). Q4 is smallest and fastest but lower quality. Q8 is closest to the original but requires more resources. The default pulls are usually Q4, which is fine for most tasks.

Increase context window: By default, Ollama uses a 2048-token context. For longer documents:

ollama run llama3.1:8b --num-ctx 8192

Run multiple models: Ollama can serve multiple models simultaneously. Just make requests to different model names and it handles loading and unloading automatically.

Monitor performance: Check which models are currently loaded:

ollama ps

Use GPU acceleration: Ollama automatically detects and uses your GPU (NVIDIA via CUDA, Apple Silicon via Metal). If you want to force CPU-only mode:

OLLAMA_NO_GPU=1 ollama run llama3.1:8b

Keep models updated: Model weights get updated occasionally with fixes:

ollama pull llama3.1:8b

This re-downloads only if a newer version is available.

Common Use Cases for Developers

1. Code generation and completion: Use as a local Copilot alternative
2. Code review: Pipe diffs through a custom reviewer model
3. Documentation writing: Generate docs from code comments
4. Data transformation: Convert between formats without sending data externally
5. Testing: Generate test cases and edge cases
6. Commit messages: Pipe git diffs to generate meaningful commit messages

Troubleshooting

Model not loading: Check available RAM with free -h (Linux) or Activity Monitor (macOS). Close other applications to free memory.

Slow responses: Ensure GPU acceleration is active. On Linux, verify CUDA is installed. On macOS, Apple Silicon handles this automatically.

Port conflict: If port 11434 is in use, set a custom port:

OLLAMA_HOST=0.0.0.0:11435 ollama serve