Primer: How To Run AI Models Locally
By now, you’ve probably tried ChatGPT, Google Gemini, Microsoft Copilot, SpaceX’s Grok, or Anthropic’s Claude.ai. These are all server-hosted AI services.
But you can also run AI models on your own computer! If you’ve got the right hardware, doing so brings you privacy and an unmetered (i.e., no usage cost) way to experiment with the various models.
Hardware
Hardware matters. Running AI models locally—especially Large Language Models (LLMs) is a game of memory bandwidth and available memory capacity. While a fast CPU is nice, the “heavy lifting” almost always happens elsewhere: primarily the GPU and RAM.
There’s a lot of technical stuff to follow in this section, but if you want my quick, very simplified recommendation, get a post-2023 Macintosh running Apple Silicon, M4 or later, with as much RAM as you can afford. A Macbook Pro is great, as are Mac Studios and Mac Minis.
The GPU: The Engine of Inference
The Graphics Processing Unit (GPU) is the most critical component. Unlike general-purpose computing, AI workloads involve massive amounts of parallel matrix multiplication, which is exactly what GPUs are designed for.
- VRAM (Video RAM): The Bottleneck. This is the most important metric. The entire model (the “weights”) must ideally fit within the GPU’s dedicated memory to run at high speeds. If the model is larger than your VRAM, the system will “offload” parts to your system RAM, which causes a massive drop in performance (tokens per second).
- The Ecosystem War (CUDA vs. Others):
- NVIDIA (CUDA): Currently the industry standard. The CUDA software layer makes it easy for developers to optimize for NVIDIA hardware. Most local AI tools (like llama.cpp or AutoGPTQ) are optimized for NVIDIA first.
- Apple Silicon (Unified Memory): A unique advantage for Mac users. In Apple’s M-series chips, the CPU and GPU share the same pool of high-speed memory. This allows a Mac with 128GB of RAM to run massive models that would require multiple professional-grade NVIDIA GPUs on a PC.
- AMD/Intel: Growing in support, but often require more manual configuration (e.g., using ROCm on Linux) to reach the same level of optimization as CUDA.
The Art of Quantization (The Hardware “Cheat Code”)
Since high-VRAM GPUs are expensive, the community uses Quantization. This is the process of reducing the precision of a model’s weights (e.g., from 16-bit floating point to 4-bit integer).
- Impact: A 70B parameter model at 16-bit precision requires ~140GB of VRAM. At 4-bit quantization, it requires only ~40GB.
- Hardware Takeaway: Quantization allows us to run “smarter” models on “smaller” consumer hardware, though there is a slight trade-off in intelligence/accuracy.
The CPU and System RAM
While the GPU does the math, the CPU acts as the conductor.
- The Role of CPU: The CPU handles the logic of the application, the text tokenization, and managing the flow of data between the storage and the GPU.
- System RAM (The Safety Net): If your GPU lacks enough VRAM, you can use “System RAM” via technologies like GGUF (used by llama.cpp). It’s much slower than VRAM, but it’s the only way to run massive models on hardware without professional-grade GPUs.
NPU (Neural Processing Units): The New Frontier
Modern laptops (especially the latest Intel Core Ultra, AMD Ryzen AI, and Apple M-series) now include dedicated NPUs.
- The Purpose: NPUs are designed to handle low-power, continuous AI tasks (like background blur in video calls or voice recognition) extremely efficiently, saving your GPU and CPU for more intensive tasks.
- Current State: While growing, software support for NPUs in the open-source local AI community is still catching up to the dominance of CUDA.
Hardware Summary
| Hardware Tier | Typical Use Case | Primary Advantage | Primary Constraint |
|---|---|---|---|
| Consumer (NVIDIA RTX) | Fast, efficient, easy setup. | Huge software support (CUDA). | Limited VRAM (usually 8GB–24GB). |
| Workstation (Mac Studio/Pro) | Large models (70B+ parameters). | Unified Memory (huge capacity). | Higher cost; less “raw” speed than top GPUs. |
| Pro/Server (A100/H100) | Enterprise-grade, massive models. | Massive VRAM and bandwidth. | Extremely expensive; requires specialized cooling/power. |
Models
An AI model is like a digital brain that has been trained on massive amounts of data to recognize patterns and perform tasks, such as generating text, identifying images, or solving problems. There are many, many models, with new ones being added every day.
Major AI Model Families
| Model Family | Developer | Type | Primary Strength / Note |
|---|---|---|---|
| Llama (e.g., Llama 3.1) | Meta (Silicon Valley) | Open Weights | The “industry standard” for local use; massive community support. |
| Qwen (e.g., Qwen 2.5) | Alibaba Cloud (China) | Open Weights | Exceptional at math, coding, and multilingual tasks. |
| Mistral / Mixtral | Mistral AI (Paris FR) | Open Weights | Highly efficient; popular for “Mixture of Experts” (MoE) architectures. |
| Gemma | Google (Silicon Valley) | Open Weights | Lightweight and high-performing; built from the same tech as Gemini. |
| Phi (e.g., Phi-3/4) | Microsoft (Redmond) | Open Weights | “Small Language Models” (SLMs) that pack huge power into tiny sizes. |
| DeepSeek | DeepSeek (China) | Open Weights | Highly regarded for specialized coding and logical reasoning tasks. |
| Yi | 01.AI (China) | Open Weights | Known for strong performance and long context windows. |
| GPT (e.g., GPT-4o) | OpenAI (Silicon Valley) | Closed (API) | The current benchmark for general intelligence; cannot be run locally. |
| Claude | Anthropic (Silicon Valley) | Closed (API) | Renowned for nuanced writing and high-level reasoning; cannot be run locally. |
| Gemini | Google (Silicon Valley) | Closed (API) | Google’s flagship multimodal model; cannot be run locally. |
A central place has emerged as the main way to browse the various models. Have you heard of “Hugging Face“? It’s named after the whimsical “hugging face” emoji:
What Hugging Face is
Hugging Face (huggingface.co) is a hosting site for AI models, plus the tooling around them. Think GitHub, but for model weights, datasets, and demo apps.
Three parts matter:
- The Hub: at this writing, there are about 2 million model repositories. Each repo holds weight files, a config, and a model card (the README).
- Libraries:
transformers,diffusers,datasets. Python code that loads and runs those files. - Spaces: small hosted web demos.
Hugging Face does not make most of the models. It stores what Meta, Alibaba (Qwen), Google (Gemma), Mistral, DeepSeek, and thousands of individuals publish.
One caution on the word “open”. Most of these are open weight, not open source. You get the weights and a license. You do not get the training data. Licenses vary, so read them before commercial use.
Can you run them locally?
Yes, for many of the models on Hugging Face. The limit is memory, not permission.
A rough rule: a model needs about 0.6 GB of RAM or VRAM per billion parameters at 4-bit quantization. Quantization compresses the weights and costs a little quality.
| Your memory | What runs well |
|---|---|
| 8 GB | Qwen3 8B, Gemma 3 4B |
| 16 GB | Qwen3 14B, Gemma 3 12B, gpt-oss 20B |
| 24 to 32 GB | Qwen3 30B-A3B, Qwen3-Coder 30B |
| 48 GB+ | Llama 3.3 70B, gpt-oss 120B |
Apple Silicon Macs are good at this. The unified memory means a 64 GB MacBook Pro can hold models that need a very expensive NVIDIA card otherwise.
The largest frontier models (DeepSeek at 600B+ parameters) do not fit on a laptop. They need a server rack.
Is there a standard runtime?
There’s no single standard. But the field has settled into three clear lanes, and each lane has one dominant tool.
llama.cpp is the local standard. It is a C/C++ engine that runs on almost anything, from a Raspberry Pi to a Mac. It reads a file format called GGUF, which packs the weights and the metadata into one file. Ollama, LM Studio, and GPT4All are all friendlier wrappers around llama.cpp. If you download a .gguf file, it will run on one of these platforms.
vLLM is the server standard. It runs on Linux with NVIDIA or AMD GPUs, handles many users at once, and gives you an OpenAI-compatible API. Most commercial providers of open models run vLLM underneath. It reads the raw safetensors files straight from Hugging Face.
MLX is Apple’s framework and is now the fastest path on M-series chips, often 2 to 3 times quicker than the older Metal backend.
So the formats have standardized more than the engines have. Safetensors is the universal storage format on the Hub. GGUF is the universal format for local, quantized use. Almost everything reads one or the other, and converting between them is routine.
One Great Place To Start: LM Studio
Have a Mac Mini or Macbook Pro? A lot of people like LM Studio.
It gives you a search box, a download button, and a chat window. Pick Qwen3 14B or Gemma 3 12B and see how it feels.
LM Studio is a very nice experience. Simply download the free version, then choose a model. I have a high-end Macbook Pro (128Gb Ram, Apple M5 Max), and the Gemma 26 model is a great choice.
LM Studio presents you with a very simple chat interface (just like hosted ChatGPT or Grok)… and the software makes it very intuitive to download a model from Hugging Face and use it:
Ollama Quick Start
If you like the terminal / command line better, as I often do when I’m just starting out (to get the foundations right), use Ollama instead:
ollama run qwen3:14b.
Step 1: Install Ollama
Ollama is an engine that can run many models on your machine. It is free, and open source.
- Go to ollama.com/download.
- Click the button for your system.
- On a Mac, open the downloaded file and drag Ollama into your Applications folder. On Windows, run
OllamaSetup.exe. You do not need an admin password. - Launch it once.
Ollama now runs quietly in the background. Look for a small llama icon in your menu bar (Mac) or system tray (Windows). It stays there.
Step 2: Open a terminal
The terminal is a window where you type commands instead of clicking buttons.
- Mac: Press Command and Space together. Type
Terminal. Press Return. - Windows: Press the Windows key. Type
PowerShell. Press Return.
A plain window opens with a blinking cursor. This is your terminal.
Step 3: Confirm the install worked
Type this and press Return:
ollama --versionYou should see a version number. If you see “command not found”, close the terminal, open a new one, and try again.
Step 4: Download and start a model
Type this and press Return:
ollama run qwen3:8bCode language: CSS (css)Two things happen. First, Ollama downloads the model. This is about 5 GB, so it takes a few minutes on a normal connection. You will see a progress bar.
Then the download finishes and you see a prompt like this:
>>>The model is now loaded into your computer’s memory and waiting.
Step 5: Ask it something
Type a question and press Return. For example:
Write three subject lines for a newsletter about weekend hiking trails.The answer prints a few words at a time. That is the model thinking, one token at a time.
Nothing you type leaves your computer. Turn off your wifi and it still works.
Step 6: Stop
Type /bye and press Return. You are back at the normal terminal prompt.
To use it again later, type ollama run qwen3:8b again. The model is already downloaded, so it starts in seconds.
Which model should you start with?
Model names follow a pattern: name:size. The b means billion parameters. More parameters usually means a smarter model and a slower one.
| Your memory | Try this | Download size |
|---|---|---|
| 8 GB | llama3.2:3b | about 2 GB |
| 16 GB | qwen3:8b | about 5 GB |
| 16 GB | gemma3:12b | about 8 GB |
| 32 GB | qwen3:14b | about 9 GB |
| 48 GB or more | qwen3:30b | about 18 GB |
Start small. If it feels fast, try the next size up. Browse the full list at ollama.com/library.
Ollama commands worth knowing
| Command | What it does |
|---|---|
ollama list | Shows every model you have downloaded |
ollama ps | Shows what is running right now |
ollama rm qwen3:8b | Deletes a model and frees the disk space |
ollama pull gemma3:12b | Downloads a model without starting it |
ollama stop qwen3:8b | Unloads a model from memory |
What to expect
Be realistic. A local model on a laptop is not the same product as a hosted frontier model.
- Speed: Expect 10 to 40 words per second on a modern Mac. Slower on an older PC without a graphics card.
- Quality: A good 8B model in 2026 is roughly as capable as a leading hosted model from two years ago. That is genuinely useful for drafting, summarizing, and rewriting.
- No internet: The model cannot search the web or read a link. It only knows what it learned during training.
- Heat: Your fans will spin up. Your battery will drain faster.
If something goes wrong
It answers one word per second. The model is too big for your machine. Delete it with ollama rm and try a smaller one.
“Error: model requires more system memory”. Same problem. Pick a smaller model.
Your disk is full. Run ollama list to see what you have downloaded, then ollama rm the ones you do not use.
The terminal says “command not found”. Ollama did not install correctly, or you need to open a fresh terminal window. Try the restart first.
Now go have some fun!
Sources:
